I took this for a spin, and I'm still seeing the two errors as before:

1. If a source or snapshot doesn't have a database/target_database defined that's equal to source_name/schema/target_schema. In the example below, dbt_default is the target.schema configured inprofiles.yml, and dbt_snapshot is configured from within the snapshot block.

Error while parsing relation test_ts:
      identifier: test_ts
      schema: dbt_snapshot
      database: dbt_default
  On Spark, database should not be set. Use the schema config to set a custom schema/database for this relation.

2. After manually setting the database/target_database param, I see the following when trying to generate docs:

Encountered an error while generating catalog: Compilation Error
  Expected only one schema in spark _get_one_catalog
dbt encountered 1 failure while writing the catalog

I think we still need a way to:

• Set database = None for all nodes/relations, including sources and snapshots
• Tell dbt to _get_one_catalog per unique schema instead of per unique database

I checked out the latest code in this branch, and I'm still seeing similar issues as above.

Sources + snapshots

The first error case now returns Cannot set database in spark! instead of On Spark, database should not be set.... To replicate:

• Add a source with a name (or schema) that is different from target.schema (as configured in the profile), and missing a database param
• Or, as above but a snapshot with target_schema that is different from target.schema (from profile), and missing target_database
• Run dbt compile

The way to resolve the error currently is by setting the database config to the same value as the schema. This double-setting shouldn't be necessary, and it's counterintuitive because we don't want users to use the database configs at all.

Catalog generation

The second error case is still Expected only one schema in spark _get_one_catalog. To replicate:

• Add a model with a custom schema configured.
• Run dbt docs generate

Somehow, dbt needs to know to run _get_one_catalog once per schema instead of once per database (which should now always be None). It looks like the run-once-per-database behavior is implicit within the information_schema argument... is there a way around this?

Next steps?

Ideally, the scenarios above would be included in our integration testing suite, which we know is a ways away from where it needs to be. If it would be helpful in continuing work on this branch, I can try adding to the dbt-integration-tests spark branch:

• a source
• a model with a custom schema

I think the highest-priority next step is figuring out if these are fixes we can make within the plugin, or if there is a lingering core issue in the way.

I've added fixes to the issues you identified.

It looks like the run-once-per-database behavior is implicit within the information_schema argument... is there a way around this?

This happens in the adapter portion of the code, so yup!